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1. Introduction 


Education has been at the centre of anti-poverty and pro-growth policies both in the 
developing world as well as in wealthier countries. It is seen as a key to development 
and to the ability of a country to keep up with the fast moving technological change 1 . 
The recent increase in the returns to education in the US and the UK has reinforced 
this view. Education is also seen as a way for individuals to escape poverty and 
welfare (and possibly crime) dependency and this perception has motivated numerous 
policies worldwide that promote education as a long run solution to these problems. 

The most recent figures (OECD, 2004) shows that in a league table of 30 developed 
OECD countries the US has slipped down the tables from 1 st to 10 th place with 87 per 
cent of 25-34 year olds having at least upper secondary education in 2002. 4 Elsewhere 
the UK has fallen to 22 nd place, down from 13 th place just a generation ago 5 , with just 
70 per cent of 25-34 year olds with upper secondary education as of 2001 which is 17 
percentage points lower than the corresponding figures for the US despite continuing 
problems with drop-out rates in some US cities 6 . This compares with over 90 per cent 
in countries such as the Japan, Korea, and Sweden: thus the proportion of youngsters 


1 See among many others Benhabib and Spiegel (1994), Krueger and Lindahl (2001) and Vandenbussche, Aghion and 
Meghir (2004). 

2 |uhn, Murphy and Pierce (1995). 

3 Gosling, Machin and Meghir (2000). 

4 In the US, 84 per cent of 55-64 year olds had at least upper secondary education in 2002 which is the highest of all 30 
countries covered. 

5 By a generation ago we look at where the UK stood in terms of the percentage of 55 to 64 year olds with at least 
upper secondary education in 2001. 56 per cent of UK 55-64 year olds had at least upper secondary education in 2002 
which places the UK 13 th out of the 30 countries covered — see OECD (2004), Table A2.2. 

6 In the US, students may drop out of school if they have reached the age set in their state’s law for the end of 
compulsory schooling which ranges between 16 and 18, but dropouts are not considered to have completed school 
and no certificate or award is issued at this stage. The U.S. dropout rate is just over 11 per cent of secondary-level 
students aged 16 and older according to the latest US Department of Education figures (see http://www.ed.gov). 
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dropping out of school at the age of 16 and failing to obtain upper secondary 
education qualifications in the UK is very high compared to most developed 
countries. 

There has been worldwide focus on school dropout problems and a number of policies 
devised to help reduce school dropout rates. One of the key policy changes in most 
OECD countries after World War II was to introduce free secondary school education 
and to increase the compulsory school leaving age. The timing and pace of these 
reforms varied tremendously across countries and in the US the most important 
reforms actually occurred before the Second World War (see Goldin, 1999). In the 
UK fees for state secondary schools were abolished by the Education Act 1944 (The 
Butler Act) and the compulsory school leaving ages was increased from 14 to 15 in 
1946 and then from 15 to 16 in 1974 where it remains today. In the US today, the 
compulsory school leaving age ranges from 16 to 18 and for the remaining for 28 

o 

OECD countries ranges from 14 to 18 . 

Making secondary education free and increasing the compulsory school leaving age 
had an effect on school dropout and completion rates and a number of these reforms 
have been analysed in previous research. 7 8 9 In recent years a number of countries, both 
developed and developing, have introduced means-tested grants in an attempt to 
encourage students to stay in school, rather than simply raising the compulsory school 


7 Compulsory schooling ends by law at age 16 in 30 states, at age 17 in nine states, and at age 18 in 11 states plus the 
District of Columbia. Source: US Department for Education. 

8 See OECD (2004), Table C.1.2. 

9 See for example Goldin (1999) who examines the 1910 to 1940 reforms in the US, Harmon and Walker (1995) who 
exploit the changes in the compulsory school leaving age in Britain to estimate the returns to schooling and Meghir 
and Palme (2005) who exploit changes in the Swedish Secondary Education system to estimate the returns to 
education. 
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leaving age. 10 The available evidence on the importance monetary incentives for 
educational participation originates mainly from direct modelling of individual 
choices in the presence of alternative tuition levels as in Heckman, Lochner and Taber 
(1999), who also consider the general equilibrium effects of varying such subsidies. 
Dynarski (2003) examines the impact of incentives for College attendance and 
completion in the US. There is however little direct evidence on the importance of 
monetary incentives for school participation. The most prominent large-scale example 
in this line of work is the use of the evaluation data from the PROGRESA program in 
Mexico (see Schultz, 2003). Todd and Wolpin (2003) use the PROGRESA data to test 
whether using a model with the wage as the opportunity cost of schooling is capable 
of predicting the impact of a schooling subsidy. They get mixed results. Attanasio, 
Meghir and Santiago (2005) test directly whether the impact of a wage reduction is 
equal to that of introducing the subsidy and reject the hypothesis. These results 
highlight the importance of direct evidence of the impact of subsidies for policy 
evaluation. Another interesting recent example is by Angrist and Lavy (2004) who 
use a randomised experiment to assess the sensitivity to monetary incentives for 
obtaining a high school graduation certificate in Israel. 

This paper examines the impact of a program that subsidizes children to remain in 
school for up to two years beyond the statutory age. The programme was first piloted 
in a number of areas in England from September 1999. Evaluating such interventions 
is of course critical to the shaping of education policy and the effectiveness or 


10 Prominent examples are the AUSTUDY program introduced in Australia in 1988 for children in their last 2 years of 
secondary school (now called YOUTH ALLOWANCE) (see Dearden and Heath, 1996), the PROGRESA program in 
Mexico which covers children from primary school to the end of high school (see Schultz, 2000, Attanasio, Meghir 
and Santiago, 2005), and the recendy introduced Familias en Action program in Colombia modelled on PROGRESA 
(Attanasio et al. 2005). 
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otherwise of a conditional cash transfer to 16 and 17 year olds on school dropout rates 
is of general policy interest to policy makers worldwide 11 . 

The presumption of the policy makers has been that these low levels of education are 
due to financial constraints rather than to the outcome of an informed choice in an 
unconstrained environment . The evaluation of this programme cannot provide 
information on the importance of liquidity constraints on education, since it changes 

i o 

the relative costs of remaining in school . However, it can provide valuable 
information on whether such subsidies, which effectively reduce the cost of 
education, actually reduce school dropout rates, which at present is the central policy 

14 

concern . 

We find that the impact of the subsidy is quite substantial, especially for those who 
receive the maximum payment. The subsidy increases the initial education 
participation of eligible males by 4.8 percentage points and eligible females by 4.2 
percentage points. In the second year this increases to 7.6 percentage points for 
eligible males and 5.3 percentage points for females, suggesting that the effect of the 
policy is not only to increase participation, but also retention in full-time education. 
The initial effects are largest for those who receive the maximum payment although 


11 There is already evidence that financial aid paid to college students has a significant impact on college attendance 
and completion. See for example Dynarski (2003). 

12 “We recognise that for some young people there are financial barriers to participating in education, particularly for 
those from lower income households.” Department for Education and Skills, General Information about EMA., 
http:/ /info.emasvsl .dfes.gov.uk/ control.asp?region=partners&page=general . 

13 Some papers that have looked at this question include Cameron and Heckman (1998), Carneiro and 
Heckman (2002), Cameron and Taber (2000), Dale and Krueger (1999) and Keane and Wolpin (1997) amongst others. 

14 With respect to dropping out at 16, following the GCSE qualification which is obtained at that age, the then minister 
for Lifelong Learning Margaret Hodge stated in Parliament: “The Real challenge is to increase the number of young 
people achieving two A-levels. That comes under our schools agenda-our 14-19 agenda. A particular problem is the 
haemorrhaging of young people, who achieve five A to Cs at GCSE level and then do not stay on to do further 
education full time”, House of Commons Hansard Debates for 5 July 2001 (pt 3). A recent survey of government policy by 
Johnson (2004) also highlights this concern. 
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the retention effects are concentrated among individuals who are only partially 
eligible. We estimate that just over half of individuals who stayed in education were 
drawn from inactivity rather than work. The overall impact of the EMA was not 
diminished when it was paid to the mother rather than to the child, though there is 
some weak evidence that paying to the child is more effective for those fully eligible 
whereas the opposite is true for those who are partially eligible. 

We also find that the effect of EMA is largest for children coming from a poorer 
socio-economic background. Both girls and boys coming from low-income families 
who qualify for the full EMA payment have very high drop out rates and the EMA has 
proved especially effective in plugging the dropout gap for this vulnerable group. 

The paper proceeds as follows. In section 2 we describe the programme and its 
variants and describe the data we use to evaluate the program. In section 3 we discuss 
the evaluation methodology and in section 4 we discuss the results. In section 5 we 
offer some concluding remarks. 

2. Background and Data 

The Education Maintenance Allowance (EMA) pilots were launched in September 
1999 in 10 Local Education Authorities. The scheme paid a means-tested benefit to 
16-18 year-olds who remained in full-time education after year 11, when education 
ceases to be compulsory (i.e. after 16 years of age approximately). The payments 
consisted of a weekly allowance (during term time only), a retention bonus every term 
for those attending consistently and an achievement bonus paid at the end of the 
course if students fulfilled goals set out in a learning agreement agreed signed by 
parents and students when they first start receiving EMA. The benefit could be 
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claimed for up to 2 years (or 3 for young people with special educational needs) and 
could be used to attend any form of full time post- 16 education, whether academic or 
vocational. In this paper we look at the effects of the EMA on individuals who first 
became eligible for the EMA in September 1999 15 . 


Table 2.1. The Four Variants of EMA 


Variant 

Maximum 

Weekly 

Retention bonus 

Achievement 


weekly EMA 
award 

payment paid to 

(per term) 

bonus 

1 

£30 

Young person 

£50 

£50 

2 

£40 

Young person 

£50 

£50 

3 

£30 

Parent 

£50 

£50 

4 

£30 

Young person 

£80 

£140 


Four different variants of the EMA were piloted and these are outlined in Table 2.1. 
In this paper we look at the overall impact of EMA and whether this impact varied 
according to either the generosity of the scheme and/or to whom it was paid (parent or 
child). The basic EMA variant 1 was piloted in 3 urban areas and 1 rural area. 
Variants 2, 3 and 4 were all piloted in 2 urban areas. 

In each area the maximum EMA weekly payment (£30 or £40) could be received by 
young people whose parents’ incomes were £13,000 or below 16 . The benefit was 
tapered linearly for family incomes between £13,000 and £30,000 with those from 
families earning £30,000 receiving £5 per week. No payment was made for families 
with income in excess of £30,000. In addition at the end of a term of regular 


15 We also have data on a second cohort who became eligible for the payment from September 2000. We have not 
included this cohort in our analysis as there is a chance that their academic outcomes in Year 11 may have been 
influenced by the announcement of the program whereas this was not true for the first cohort because of the timing of 
the announcement. We concentrate on urban areas only as it was only in urban areas that all 4 variants were piloted. 
Full results for all cohorts and individuals who participated in the pilots are available from the authors. 

16 Income is defined as the taxable income of the biological parents in the previous tax year. 


7 



17 

attendance the child would receive a non-means-tested retention bonus (£50 or £80) . 
The children also received an achievement bonus on successful completion of the 
course examination. To put these amounts in context the median net wage among 
those who opted for full-time work in our sample was £100 per week, corresponding 
to less than 40 hours’ work a week. Thus the maximum eligibility for the EMA, 
depending on the variant, replaces around a third of post tax earnings. 

The programme was announced in the spring of 1999, just before the end of the 
school year and the lateness of the announcement means that it could not have 

1 Q 

impacted on a child’s Year 11 examination results . The data used to evaluate the 
programme are based on initial face-to-face interviews with both the parents and the 
children and follow up annual telephone interviews with the children. The data set 
was constructed so as to include both eligible and ineligible individuals in pilot and 
control areas 19 . The first interview was conducted at the beginning of the school year 
in which the subsidy became available. In the following year the same students (but 
not parents) were followed up using a telephone interview. 

We collected a wealth of variables relating to family income and background, 
childhood events (such as ill health and mobility), prior school achievement as well as 
administrative data on the quality of schooling in the child’s neighbourhood as well as 


17 This bonus was paid to the child in ALL variants (including variant 3). 

18 This was not true for our second cohort and for this reason they are excluded from the analysis. We feel that it is 
important to control for student ability and the only measures we have relate to school outcomes in Year 11. 

19 We used data from the British Youth Cohort Studies to choose our control areas so as to ensure the background 
characteristics of the control areas in terms of historical education participation, background characteristics of parents 
and neighbourhood characteristics were as similar as possible to those of the selected pilot areas which we knew in 
advance. 



other measures of neighbourhood quality measured prior to the introduction of the 
EMA 20 

Table 2.2 provides some pre-reform neighbourhood statistics for our pilot and control 
areas, while Appendix 1 provides definitions of each of these neighbourhood 
variables (which are based on government benefit figures and are produced annually 
by Oxford University). Larger values of these indices point to a greater level of 
deprivation. For the sake of comparison we also show the average indices and their 
standard deviation for the whole of England. Based on this it is clear that the pilots 
and controls are in more deprived areas and remarkably close to each other relative to 
the overall variation in England. As can be seen from the table, the characteristics of 
the treatment and control areas are very similar indeed, with pilot areas tending to be 
slightly more disadvantaged. Indeed the (proxy for the) aggregate non-school 
participation rate pre-reform is just less than 3 percentage points higher in our pilot 
areas than our control areas. This highlights the importance of appropriately 
weighting our control group as if we do not take this pre-reform difference into 
account we are likely to underestimate the EMA impact. 

To control for differences between pilot and control areas we also use individual level 
data from our survey as well as this administrative and local area data. The variables 
we use include individual based characteristics on prior achievement, household 
income, parental occupation and education, household composition, ethnicity; 
childhood variables on early health problems, early childcare and grandparental 
inputs, special needs, and geographic mobility in early life. We have also matched on 


20 The neighbourhood data we used was based at ward level which can cover as few as 400 people to as many as 
30,000 people, but usually between 5,000 to 7,000 individuals. There are just under 9,000 wards in England and Wales. 
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publicly available data on the pre-reform quality of the child’s nearest Year 11 state 
school 21 and distance to the nearest state year 12 educational provider (post 16 
education) . Summary Statistics for our remaining variables used in the analysis are 
provided in Appendix 2. 


21 We have address grid references for every child in our survey as well as for every Year 11 school in the country. This 
allowed us to identify the nearest (as the crow flies) comprehensive Year 11 school for every child in our survey. Once 
we identified the school, we mapped in publicly available pre-reform quality measures from that school. 

22 A number of studies have shown that distance to school is an important determinant of educational decisions (see 
Card, 1995 and 1999). 
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Table 2.2. Pre-reform neighbourhood characteristics of pilot and control areas 



Pilot Areas 


Control Areas 

All England 



Mean 

Std 

Mean 

Std 

Mean 

Std 



Dev 


Dev 

Dev 

Number of observations 

4,518 


2,320 

8,414 


Measures of local deprivation 







(index) 







Multiple deprivation score 1 

38.36 

17.00 

37.05 

18.64 

21.70 

15.39 

Income 1 

30.14 

11.79 

29.93 

13.62 

18.86 

11.31 

Employment 1 

16.66 

6.23 

16.38 

7.14 

10.19 

6.49 

Health Deprivation and Disability 1 

1.04 

0.58 

0.97 

0.68 

0.00 

0.92 

Education, Skills and Training 1 

0.78 

0.98 

0.70 

0.82 

0.00 

0.87 

Housing 1 

0.47 

0.83 

0.34 

1.00 

0.00 

0.92 

Geographical Access to Services 1 

-0.53 

0.46 

-0.60 

0.54 

0.00 

0.87 

Child poverty 1 

43.78 

17.12 

42.70 

19.61 

26.74 

17.02 

Education drop-out rates 1998 







Per cent dropping out of school at 

38.9 


35.8 


30.0 


Per cent not staying on at school 5 

69.15 

8.99 

66.63 

10.50 



Nearest school data 







Class sizes 1999 

21.43 

2.29 

21.41 

2.23 



Authorised absences (per year) 

8.69 

1.99 

8.86 

2.63 



% getting 5 GCSE 2 A-C in 1999 

35.35 

17.72 

35.48 

15.82 



% getting 0 GCSE 2 A-G in 1999 

6.93 

5.09 

6.54 

6.08 



School has 6th form 3 ? 

0.45 

0.50 

0.34 

0.47 



Distance to nearest vear 12 provider 

1.630.7 

1051.1 

1.951.6 

1.480.8 




1 A higher score indicates a higher incidence of deprivation. Scores across different measures are not 
comparable. 2 GCSE exams are taken in the last year of compulsory education (Year 11) and are 
graded A to G. 3 The 6 th form is the two years of post-compulsory schooling. Years 12 and 13. The all 
England data is calculated on the basis of ward level data (small subdivisions of municipalities). There 
are 8,414 wards in England. 4 This data is taken from official LEA based calculations of 16 year old 
stay-on rates in 1998 (see Department for Education and Skills (2005)), weighted by our sample 
populations 23 . 5 This data is calculated by looking at the number of 17,18 and 19 year-olds in receipt of 
child benefit divided by the number of 13, 14 and 15 year-olds receiving the benefit in the local area 
(ward). Child benefit is payable for all children under 16 and all those over 16 in secondary education. 
It has nearly 100% take-up. As very few 19 year-olds are in secondary - rather than tertiary - 
education, this figure is an underestimate (by about 1/3) of the proportion of young people staying in 
post-compulsory education and should be understood as a proxy for this figure. 


3. The evaluation Methodology - Matching 

The outcome of interest in this paper will be participation in post-compulsory school, 
i.e. in Year 12 and Year 13. As we discuss in the results section below, we are 
interested in the impact of financial incentives on the entire target population, on the 


23 This is necessary as in 2 of our control LEAs we sampled half as many individuals as in our other control LEAs. 
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population of those partially eligible for the subsidy and on the ineligible population. 
In each case we will be comparing the outcomes relative to the appropriate 
comparison group. Although the treatment and control areas are very well matched, 
the distribution of characteristics is not identical, as they may have been following a 
successful and large-scale randomisation. To allow for the fact that this was not going 
to be a randomised experiment, we have collected a large array of individual and local 
area characteristics, which should control for any relevant differences in the treatment 
and control areas before the program was introduced. 

The method we use to balance the distribution of observable characteristics is 
propensity score matching. We provide a brief description in Appendix 5' . It turns 
out that a simple fully interacted OLS model imposing common support gives almost 
identical results to our preferred matching estimator, confirming the findings of 
Blundell, Dearden and Sianesi (2005). 

As a final step we also carry out some sensitivity analysis using difference in 
differences based on aggregate data and on the behaviour of older siblings. In the 
former case we consider aggregate school participation data for 16 year olds including 

oc 

eligible and ineligible pupils” . In the latter case we compare the change in school 
participation between the younger and the older sibling in pilot and control areas. In 
doing this we also control for a number of characteristics. The reason this is not our 
main evaluation method is that not all children have older siblings of the same gender 
and secondly the time varying covariates we measure, including income, relate to the 
date of the survey, i.e. when the younger sibling was deciding whether to continue in 


24 Rosenbaum and Rubin (1983), Heckman, Ichimura and Todd (1994). 

25 See Department for Education and Skills (2005). 
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education or drop out. Nevertheless, this sensitivity analysis confirms the results we 
find with matching. 

In all cases the standard errors are computed using the block bootstrap with a cluster 
being defined as a school. 

4. The results 

4.1 Impact of the EMA on Year 12 Destinations 

Table 4.1 shows estimates of the impact of the EMA (overall and by gender) on 
young people’s initial decisions to remain in full-time education, to move into 
employment or to be inactive (NEET - Not in Education, Employment or Training). 

For the purposes of demonstrating robustness across econometric methods we present 
results based on comparing simple means (unmatched), linear OLS, OLS with 
interactions for heterogeneous effects (fully interacted linear matching) and non- 
parametric kernel based matching 26 . All give very similar results and in our 
discussion from now on we focus on fully interacted linear matching allowing always 
for common support. 

The EMA has had a positive and significant effect on post-compulsory education 
participation among eligible young people. The overall estimate is 4.5 percentage 
points from a baseline of 64.7 per cent in our matched sample of controls . This 


26 Our preferred matching estimate uses an Epanechnikov kernel with a bandwidth of 0.06. We tested a number of 
different methods of matching including Epanechnikov kernels with a variety of bandwidths, nearest neighbour 
matching, and Mahalanobis-metric matching method and based our decision on which method gave us the best 
covariance balancing indicators. In all cased our preferred matching estimator gave the best results in terms of various 
covariance balancing measures (see Appendix 3). 

27 The baseline figure is different from the aggregate figure for a number of reasons. First the population is different. 
Second, the age window that the aggregate figure looks at is different since the aggregate figures work with age and not 
with school years as we do. Thus the aggregate figures relate to slightly older persons. Finally, we may have had 
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increase has drawn young people from both employment and the inactivity group 
(NEET) in equal parts in the urban areas. This is significant because it shows that to a 
large extent the policy is not displacing individuals from work, but from unproductive 
activities, thus implying an overall lower cost of providing this incentive to education. 
This does raise the issue of the quality of individuals attracted to education from the 
subsidy, since it seems to consist largely of individuals with little opportunity cost. 
However, as we shall see they tend to stay in full time education for the whole two 
years of the subsidy. Moreover, given the regulated nature of the education 
institutions the have to attend one can hypothesize they are receiving valuable 
training. Ultimately however this can only be evaluated using eventual labour market 
outcomes, not available to us. 


differential non-response between participants and non-participants. Note however that there is no evidence that the 
non-response is different between pilots and controls. In fact the results on attrition imply that any non-response will 
be balanced between pilots and controls. 
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Table 4.1. Impact of EMA on Year 12 destinations of eligibles 



Participation 

in 

Pilot Areas 

Unmatched 

OLS 

Fully 

Interacted 

OLS 

Matching 

Estimate 

Total: 

FT Education 

69.2 

3.9 

3.8 

4.5 

4.5 

(S.E) 


(1.4) 

(1-3) 

(1-4) 

(2.3) 

Work/Training 

16.4 

-0.4 

-1.0 

-1.7 

-1.7 

(S.E) 


(1.1) 

(1.1) 

(1-1) 

(2.4) 

NEET 

14.5 

-3.5 

-2.8 

-2.7 

-2.7 

(S.E) 


(1.1) 

(0.9) 

(1-0) 

(2.0) 

Sample size: 


5,315 

5,315 

5,299 

5,299 

Males: 

FT Education 

66.4 

5.3 

4.8 

5.0 

4.8 

(S.E) 


(2.0) 

(2.0) 

(2.1) 

(2.3) 

Work 

19.7 

-1.5 

-2.1 

-2.5 

-2.9 

(S.E) 


(1.7 

(1-8) 

(1.8) 

(2.0) 

NEET 

13.9 

-3.8 

-2.7 

-2.4 

-1.8 

(S.E) 


(1.5) 

(1.5) 

(1.5) 

(1.7) 

Sample size: 


2,653 

2,653 

2,643 

2,643 

Females: 

FT Education 

71.9 

2.5 

2.9 

4.0 

4.2 

(S.E) 


(1-9) 

(1-7) 

(1-8) 

(2.3) 

Work 

13.0 

0.7 

0.4 

-0.4 

-0.5 

(S.E) 


(1-4) 

(1-4) 

(1-5) 

(2.0) 

NEET 

15.1 

-3.2 

-3.3 

-3.6 

-3.6 

(S.E) 


(1.6) 

(1.2) 

(1.4) 

(1.7) 

Sample size: 


2,662 

2,662 

2,656 

2,656 


Notes: All standard errors allow for clustering at school level. The standard errors reported for our 
matching estimator are based on 1,000 replications and use stratification at the Local Education 
Authority level. Our fully interacted OLS model imposes common support for males and females. 


The effects are higher for males, who have lower participation rates than for women. 
However the difference is not significant. 


4.2 The longer term impact of the EMA. 

So far the analysis has concentrated on the impact of the EMA on initial destinations 
in Year 12, the first post-compulsory year. However, the EMA is designed to last for 
two years. Thus an important question is whether the impact of the EMA persists in 
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the 2 nd year, altering significantly the entire path post 16. To answer this question, we 
focus on individuals who we observe for a second year, and examine their 
destinations in Year 13, one year after the introduction of EMA. 

When considering whether the policy has led to longer term increases in participation 
we will have to use the 2 nd wave of data for our cohort. However, there has been some 
attrition. About 25 per cent of the original sample was lost in the follow up. In 
Appendix 4 we show that the likelihood of remaining in the sample is higher for those 
with incomes that would make them eligible for the EMA relative to the rest. 
However, the pattern of attrition is the same for the treatment and control areas, 
possibly implying that any biases due to attrition balance out. In Appendix 4 we 
report the results of running a probit on the determinants of attrition. We see that 
those who come from families earning less than £13,000 per annum (i.e. those in our 
pilot and control groups who we define as fully eligible) are slightly more likely to 
drop out of the panel but there is no difference conditional on this eligibility between 
pilot and control areas. These results suggest that attrition was not directly related to 
the EMA. When we re-estimate the impact of EMA in the first year only on the 
sample who do not drop out of the panel we obtain slightly lower estimates of the 
overall impact of EMA on full-time education participation with our male estimates 
being slightly but not significantly larger” and our female estimates being slightly but 
not significantly smaller . Whilst this is reassuring, it is also clear that the 
distribution of observable characteristics has changed, as a result of attrition in the 2 nd 
wave. In particular the ones who did not drop out of the sample originate from a better 


28 5.0 percentage points with a standard error of 2.7, compared to our estimate of 4.8 percentage points for the full 
sample (see Table 4.1). 

29 3.5 percentage points with a standard error of 2.4, compared to our estimate of 4.2 percentage points for the full 
sample (see Table 4.1). 
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family background and were more likely to be in school in wave 1 of the data (see 
Table 4.2 below). In this sense the population for which we will be looking at the 
longer term outcomes is different than the one for which we can look at the shorter 
term ones. However it should be stressed that issues relating to the impact of attrition 
are only relevant when we look at the longer-term effects of the program. 

We define the potential outcomes that could occur two years after the introduction of 
the program as: education in Year 12 and education in Year 13; education in Year 12 
and other activity in Year 13; other activity in Year 12 and education in Year 13; and, 
finally, other activities in both year 12 and year 13. Hence the overall impact on full- 
time education in Year 12 for this second wave can be found by comparing the 
outcomes of those in our first two groups with those in our second two groups in the 
first year. 

Table 4.2 shows the impact of EMA based on the division of the population into the 
four mutually exclusive groups described above using our preferred kernel matching 
technique. The important conclusion that comes from Table 4.2 is that where the 
EMA has been effective it has led to an increase in both year 12 and year 13 
attendance and thus it is shown to have long-term effects. This is important because it 
indicates that those drawn into education due to the EMA are committed to it. They 
do not just sample it only to find that it is not for them and to drop out a few months 
later. It also shows that the EMA has increased average education retention rates, 
defined as the proportion of those in full-time education in Year 12 who were still in 
full-time education in Year 13. EMA increased average retention rates by 4.0 
percentage points (from 77.9 per cent to 81.9 per cent), with a particularly large effect 
for men (6.1 percentage points). 
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Table 4.2: Impact of EMA on Year 12 and Year 13 destinations for Eligibles 



Participation 

Impact 


in Pilot Area 

(Kernel 




Matching) 

Male: 




Education Y12 — > Education Y13 

58.7 


8.1 

(S.E) 



(2.8) 

Education Y12 — > Other activity Y13 

13.1 


-3.1 

(S.E) 



(2.1) 

Other activity Y12 — > Education Y13 

1.7 


-0.5 

(S.E) 



(0.9) 

Other activity Y12 — > Other activity 
Y13 

26.4 


-4.5 

(S.E) 



(2.6) 

Retention Rate (for those in Edn in 
Y12) 

81.7 


6.1 

(S.E) 

Sample size 
Female: 


1211 

(3.0) 

Education Y12 — > Education Y13 

63.4 


4.4 

(S.E) 



(2.8) 

Education Y12 — > Other activity Y13 

13.8 


-0.9 

(S.E) 



(2.2) 

Other activity Y12 — > Education Y13 

2.9 


0.8 

(S.E) 



(0.8) 

Other activity Y12 — > Other activity 
Y13 

19.9 


-4.4 

(S.E) 



(2.3) 

Retention Rate (for those in Edn in 
Y12) 

82.1 


2.0 

(S.E) 

Sample size 
TOTAL: 


1295 

(2.8) 

Education Y12 — > Education Y13 

61.1 


6.2 

(S.E) 



(3.3) 

Education Y12 — > Other activity Y13 

13.5 


-2.0 

(S.E) 



(2.3) 

Other activity Y12 — > Education Y13 

2.4 


0.2 

(S.E) 



(1.1) 

Other activity Y12 — > Other activity 
Y13 

23.0 


-4.4 

(S.E) 



(2.5) 

Retention Rate (for those in Edn in 
Y12) 

81.9 


4.0 

(S.E) 

Sample size 


2506 

(3.3) 


Note: All standard errors allow for clustering at school level. The standard errors reported for our 
matching estimator are based on 1,000 replications and use stratification at the Local Education 
Authority level. 
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4.3 Impact ofEMA in Year 12 and Year 13 by Eligibility Groups 


We now turn to comparing the impact of the policy separately for those who are 
eligible for the full amount of the EMA, those who are only eligible for a fraction, 
because their parents have an income higher than £13,000. The impact between the 
two groups may be different for a number of conflicting reasons. First, because the 
subsidy is lower it may have a lower effect. Second, the individuals who receive a 
lower subsidy do so because they come from a better off background. This may make 
them more likely to go to school in the first place and thus may also affect their 
sensitivity to monetary incentives. With this design we cannot distinguish one effect 
from the other. Thus, in the results that follow we distinguish between full eligibility, 
partial eligibility and ineligibility to see if the impact of EMA differs by whether a 
person was fully or only partially eligible and to see if there were any spillovers to 
those in the ineligible group. 

Only just over 47 per cent of individuals in Cohort 1 were eligible for the maximum 
EMA payment, around 31 per cent for partial payment whilst 22 per cent were not 
eligible. All eligible individuals were entitled to the bonuses that were not means- 
tested. 

For the results presented in the following Tables and all subsequent analysis we use 

TO 

fully interacted linear matching . 


30 The point estimates from these model were always very close to our preferred kernel matching estimates but have 
greater precision. All kernel based matching estimates are available from the authors on request. 
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Table 4.3 Impact of EMA on Year 12 destination: all young people by eligibility 



Males 
Particip- 
ation in 

Pilot Area 

Impact 

Females 
Particip- 
ation in 

Pilot Area 

Impact 

All 

Particip- 
ation in 

Pilot Area 

Impact 

Fully eligible: 







FT Education 

66.0 

6.7 

68.2 

6.8 

67.1 

6.7 

(S.E) 


(2.4) 


(2.4) 


(1.7) 

Work/Training 

18.6 

-0.7 

12.5 

-2.3 

15.5 

-1.5 

(S.E) 


(2.2) 


(1.9) 


(1.5) 

NEET 

15.4 

-6.0 

19.3 

-4.4 

17.4 

-5.2 

(S.E) 


(2.1) 


(1.8) 


(1-5) 

Sample size 


1,582 


1,595 


3,177 


Taper: 

FT Education 

66.6 

4.0 

77.5 

-1.6 

72.1 

1.2 

(S.E) 

Work/Training 

21.7 

(3.6) 

-6.5 

14.1 

(2.9) 

3.6 

17.9 

(2.3) 

-1.4 

(S.E) 

NEET 

11.7 

(3.3) 

2.5 

8.4 

(2.3) 

-2.1 

10.0 

(2.0) 

0.2 

(S.E) 

Sample size 


(2.1) 

1,036 


(2.0) 

1,035 


(1.5) 

2,071 


Ineligible: 

FT Education 

75.7 

3.2 

88.5 

-2.2 

81.7 

0.7 

(S.E) 

Work/Training 

18.0 

(3.3) 

0.1 

7.4 

(2.6) 

2.5 

13.1 

(2.1) 

1.2 

(S.E) 

NEET 

6.3 

(2.9) 

-3.4 

4.2 

(1-9) 

-0.3 

5.3 

(1-8) 

-1.9 

(S.E) 

Sample Size 


(2-1) 

762 


(2.0) 

685 


(1-5) 

1,447 


Note: The standard errors reported allow for clustering at school level. 


Among those who were estimated to be eligible for a full EMA award, EMA 
increased full-time education participation in Year 12 by 6.7 percentage points. For 
those estimated to be eligible for only a partial award, the corresponding figure is 1.2 
percentage points (and not statistically significant at conventional levels). Thus the 
response of those fully eligible is larger than in the population who are facing the 
taper. A recent survey of education policy in England by Johnson (2004) has 
highlighted that one of the key aims of policies like EMA is to improve post 
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compulsory staying-on rates for children from deprived social backgrounds. The 
combination of a more generous payment and possibly their greater responsiveness to 

•5 1 

the payment points to a success of the policy in this dimension. 

Similarly for ineligible individuals the overall effect is very small (+0.7 percentage 
points), and not statistically significant at conventional levels, indicating both that the 
spillover effects in the short run are not important and reinforcing our confidence in 
the results, i.e. there is no evidence that an unobservable area effect is driving the 
results. 

4.4 Who gets the payment - Does it matter? 

Our analysis suggests that there are no significant differences in outcomes for variants 
where the child receives the payment (variants 1, 2 and 4). Increasing the generosity 
of the payment (variant 2) and the level of retention bonuses does not result in any 
significant impact on measured outcomes in Years 12 or Year 13 compared to those 
in variant 1. 

In one of the EMA variants piloted (variant 3) the payment was made to the mother 
instead of the child. There are many reasons why paying the mother could have a 
different effect. In one extreme, if the mother is not expected to pass on the benefit to 
the child, then the child will have a lower incentive to attend school. On the other 
hand, since transfers are already taking place from the parents to the child, one can 
argue that even if the benefit is given to the child it can be clawed back by the parents 
and hence whether it is paid to the child or parents it should not make much 
difference. 


He says, “The UK has a relatively low staying-on rate in full time education after age 16. Given high returns this is, 
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In order to investigate this we compare outcomes in variants 1 and variants 3 where 
the only difference in the scheme is who received the weekly payment. In order to 
ensure we are comparing like with like we use same sample of individuals from the 
control group in assessing each variant and only include those who satisfy the 
common support restrictions for both variants. We estimate the effect by sex and 
eligibility group but we only report the results by eligibility group, as this is where 
important differences are detected. The results of doing this are shown in Table 4.5 

Table 4.5 Impact of EMA on Year 12 destinations: by eligibility and receipt 



Variant 1 

Variant 3 

Fully eligible: 

FT Education 

10.3 

4.8 

(S.E) 

(2.6) 

(2.1) 

Work/Training 

-1.8 

-1.6 

(S.E) 

(2.2) 

(1.9) 

NEET 

-8.6 

-3.2 

(S.E) 

(2.1) 

(1.8) 

Sample size 

1346 

1378 

Partially Eligible: 

FT Education 

-4.0 

6.9 

(S.E) 

(2.9) 

(3.6) 

Work/Training 

2.5 

-6.6 

(S.E) 

(2.6) 

(3.0) 

NEET 

1.6 

-0.3 

(S.E) 

(2.5) 

(1.9) 

Sample size 

824 

830 


Note: All standard errors allow for clustering at school level 


If we do not distinguish by eligibility, the impact of both variants is almost identical 
(4.9 percentage points for variant 1 and 5.6 percentage points for variant 3). However, 
this appears to hide interesting differences by eligibility, some of which are 
significant at the 5% level. From Table 4.5 we see that for variant 1, where the money 


perhaps, surprising and probably economically inefficient. Given very substantial differences in staying-on rates by 
social background, it is also of concern from an equity point of view” (pp 177-178). 
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is paid directly to the child, the EMA impact is concentrated solely among those who 
are fully eligible. Participation in full-time education is increased by 10.3 percentage 
points. Almost all of this increase in participation is drawn from the NEET group. 
There is no significant full-time education impact for individuals who are partially 
eligible. 

The story is very different for the variant where the payment is made to the child’s 
mother. The impact is now spread much more evenly among all groups who are 
eligible ranging between 4.8 percentage points (fully eligible) and 6.9 percentage 
points (partially eligible) and these effects are not significantly different from each 
other. 

These finding have obvious policy interest and suggest that if the key interest is in 
increasing participation among those from the poorest backgrounds (those from 
families earning less than £13,000 per annum) then payment to the child may be 
preferred, whereas if the government is keen to impact across the whole eligibility 
distribution then payment to the mother may be more effective - at least in terms of 
initial staying on decisions . 

In Table 4.6 we look at the results for Year 13 for those who do not drop out of the 
panel from our sample. Unfortunately sample sizes are quite small which affects 
precision but we see that by the second year, the results for those who are fully 
eligible is much more similar across variants. Again, for those who receive only a 
partial payment, there appears to be a bigger retention effect, but this is only for 
variant 3. By Year 13, the only big difference between the variants is that variant 3 is 


32 The results by sex are available from the authors. 

33 The EMA since September 2004 has been rolled out nationally and all payments are made to the child. 
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more effective in increasing participation amongst those who are partially eligible. 


Table 4.6 Impact of EMA on Year 13 destinations: by eligibility and receipt 



Valiant 1 

Variant 3 

Fully eligible: 

Education Y12 — > Education Y13 

8.7 

6.7 

(S.E) 

(3.0) 

(2.7) 

Education Y12 — > Other activity Y 13 

2.5 

-0.8 

(S.E) 

(2.4) 

(2.1) 

Other activity Y12 — > Education Y13 

-1.1 

0.3 

(S.E) 

(1.2) 

(1.0) 

Other activity Y12 — > Other activity Y13 

-10.4 

-6.1 

(S.E) 

(2.9) 

(2.7) 

Sample size 

875 

619 

Partially Eligible: 

Education Y12 — > Education Y13 

2.9 

11.3 

(S.E) 

(3.4) 

(3-5) 

Education Y12 — > Other activity Y13 

-3.2 

-6.2 

(S.E) 

(2.8) 

(2.8) 

Other activity Y12 — > Education Y13 

0.9 

1.2 

(S.E) 

(0.9) 

(1.1) 

Other activity Y12 — > Other activity Y13 

-0.6 

-6.2 

(S.E) 

(3.1) 

(3-3) 

Sample size 

619 

642 


Note: All standard errors allow for clustering at school level. 


4.6 Does the impact vary by prior academic achievement? 

We have already seen that the EMA has its largest impact on kids from relatively 
poor families. Another key question is whether children with low prior academic 
achievement can be made to stay in school longer, possibly improving their skills 
before labour market entry. Thus, in Table 4.7 we present results where the sample is 
split by low and high prior achievement 34 . The EMA seems to affect primarily those 
with low prior achievement. However, this is perhaps not so surprising, given that the 


34 This is based on grades obtained in GCSE Maths and English exams that all students had to sit in Year 11. Each 
grade in these exams was given a score of 0 to 8 and then added together to obtain a score out of 16. Our high ability 
kids had a score of 10 or above and this was roughly just under half our sample. 
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post compulsory school participation rate is much higher for those with high prior 
achievement. It does point out however, that the increase in participation comes 
primarily from the lower ability group and is consistent with the earlier result 
showing that a large proportion of the increase in participation comes from those who 
would not otherwise be employed. This casts some doubt on the longer-term returns 
of the policy. 


Table 4.7 Impact of EMA on Year 12 and Year 13 destinations of eligibles: by 
prior academic achievement 



Males 


Females 



Particip 

Increase 

Particip 

Increase 


-ation in 


-ation in 



Pilot 


Pilot 



Area 


Area 


Low Prior Academic Achievement 
Education Y12 — > Education Y13 

47.4 

7.7 

51.9 

6.7 

(S.E) 


(3.1) 


(3.5) 

Education Y12 — > Other activity Y 13 

16.9 

-4.9 

18.9 

-2.6 

(S.E) 


(2.8) 


(3.3) 

Other activity Y12 — > Education Y13 

1.8 

-2.1 

4.1 

1.0 

(S.E) 


(1.1) 


(1.2) 

Other activity Y12 — > Other activity Y13 

33.8 

-0.7 

25.0 

-5.1 

(S.E) 

1,134 

(3.0) 

1,100 

(3.3) 

Sample size 





High Prior Academic Achievement 
Education Y12 — > Education Y13 

84.4 

1.6 

89.4 

2.7 

(S.E) 


(2.4) 


(2.6) 

Education Y12 — > Other activity Y13 

5.7 

0.8 

5.6 

-1.4 

(S.E) 


(1-5) 


(1.7) 

Other activity Y12 — > Education Y13 

1.6 

1.5 

0.7 

-0.2 

(S.E) 


(0.6) 


(0.8) 

Other activity Y12 — > Other activity Y13 

8.4 

-3.9 

4.3 

-1.1 

(S.E) 


(2.0) 


(1.7) 

Sample size 

1,061 


1,244 



Note: All standard errors allow for clustering at school level. 
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4.7 Sensitivity Analysis 


Aggregate data 

We now present simple difference in difference estimates based on aggregate school 
participation data for 16 year olds. We use three post policy periods compared to the 
one pre -policy period (1998) where we have a complete set of data. In reading these 
results note that the proportion of fully eligible individuals is about 47 per cent. If we 
include those partly eligible (i.e. on the taper) the proportion rises to 78 per cent. So if 
the policy had no effect on the ineligible individuals we need to multiply the effect by 
a factor of between 1.3 and 2. 

The three difference-in-difference estimates for the 1999, 2000 and 2001 are 
respectively 2.7, 2.3 and 4.7 percentage points always with 1998 as the baseline. If we 
multiply these by 1.3 we obtain respectively 3.5, 3.0 and 6.1 percentage point effects, 
which are remarkably close to the effect we obtained with matching (4.5) and 
certainly within the 95 per cent confidence interval. 

Using older siblings 

An alternative approach, which allows us to focus more closely on the group of 
interest and at the same time to control for characteristics as in our main analysis, is to 
use difference-in-differences using as a comparison group the older siblings of the 
children in our pilot and control areas. We thus compare the change in participation 
between the current cohort and that of the older sibling in the pilot and comparison 
areas, controlling for observable characteristics. We include a full set of cohort and 
area dummies. We find an EMA effect of 8.4 percentage points (with a standard error 
of 2.8), which is larger than the effect we reported above. The difference is not 
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significant at conventional levels. The smaller sample has made the estimate less 
precise, but offers support for the significant effect of the EMA. 

Finally we also carry out successive difference in differences across siblings reaching 
the statutory school leaving age before the period when the policy was in place as 
well as in the final year. We find that in all previous periods the “effect” is not 
significant and the estimate is close to zero. In the final period we obtain a positive 
and significant effect, again corroborating and strengthening our results. 

4.8 A back of the envelope costs-benefit calculation 

Based on our results that the EMA increased the percentage of individuals from 
income-eligible families completing 2 years of post compulsory education by 6.2% 
from 61.1% to 67.3%, and that half of this increase represents individuals who would 
have otherwise been in paid employment, we estimate that those brought into 
education would need to experience a real increase in future earnings of 5.6% as a 
result of the additional 2 years of education for the programme to break even, 
allowing for the opportunity cost of education. 36 If we also allow £3,000 for the 
annual extra cost of educating those who stay on in secondary education , the 


35 The standard error allows for serial correlation and cluster effects. 

36 To do this calculation we find the rate of return to education (r) which solves: 

where EMA t is the annual average EMA transfer payment 

allowing for the fact that not all are fully eligible. This is estimated to be £750 a year — £25 a week for 30 weeks plus 
£150 in bonuses, and p is the proportion eligible for the EMA (estimated to be 67.3%). /t is the increase in 
participation in education (assumed to be 6.2ppt), with half of this coming from those who would have otherwise been 
in paid employment. C t is the marginal cost of those brought into education as a result of the EMA and w t represents 
the estimated lifecycle wage profile based on the 2002—03 Yamily Resources Survey. We assume 2% a year real growth in 
future wages. R is the discount rate which is assumed to be 3%% which is the recommended discount rate in the UK 
HM Treasury Green Book (http:/ / greenbook.treasury.gov.uk/) . 

37 See Department for Education and Skills (2003), Table7. 


^ p.EMA, + (A / 2 )w t + AC, _ ^3 Arw, 

h a +r)' ‘fe 
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required return to education for the two years is 7%. Research in to the returns from 
staying on in post-compulsory education suggests that the returns are in fact 11% for 
men and 18% for women (Dearden, McGranahan and Sianesi, 2004). However, the 
returns to education for men of either low-ability or low social class (i.e. the group on 
which the EMA seems to have the largest impact) is estimated at between 6%-8% for 
these last two years, which from a financial point of view makes the programme just 
about break-even. However, there may well be other benefits to the policy; infra- 
marginal individuals may reduce hours of work and increase effort put into education; 
there may be crime reductions. These are hard to evaluate benefits but they should not 
be discounted without further research. 

5. Conclusions 

Despite a steady increase the participation in education following completion of 
compulsory schooling in England remained relatively low. The government decided 
to pilot a financial incentive scheme to encourage more pupils from lower income 
families to stay on in school - the Education Maintenance Allowance (EMA). Since 
September 2004 the EMA program has been rolled out nationally. 

In this paper we use a dataset collected by us for the purposes of evaluating the 
impact of this schooling subsidy program on school participation in England. Our 
results imply that the scope for affecting education decisions using subsidies to 
education can be substantial. More specifically, the results imply that the EMA has 
raised significantly the stay on rates past the age of 16. The initial impact is around 
4.5 percentage points while having no effect on ineligibles. Taking into account that 


38 The precise marginal cost is hard to quantify since one would want to keep quality constant. We have taken the 
average expenditure per-pupil as our measure. 
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this was a time when the labour market was particularly buoyant, these seem to be 
quite large effects, although they were achieved with a replacement rate of 33%-40% 
of average net earnings. 

The results also suggest that the impact of the EMA on participation actually 
increases in the following year. For those who get the full payment, the increased 
participation is maintained whereas for those who get partial payment, retention is 
significantly improved. This result is important because it suggests that those who are 
induced into extra education do not find the courses unexpectedly difficult or 
uninteresting and are willing to stay for the full two years of the program into 
education. Importantly, about half of the increase in school participation is due to a 
decline in inactivity, rather than work. This reduces the implicit costs of the program 
since the foregone earnings for these individuals are zero. However, this may also 
mean that the programme is attracting those with few other opportunities as also 
demonstrated by the fact that the largest effect is among those with low prior 
achievement. The key policy question here is the extent to which this extra education 
is valuable to them. 

Finally, it appears that the EMA had its largest impact on children coming from 
families from the poorest socio-economic background (based on parental income). 
This is a particular policy concern and it appears that the EMA has made important 
inroads in improving the prospects of these children. 

The results in this paper demonstrate that a conditional payment to 16 and 17 year 
olds can significantly reduce school dropout rates. Of course a number of important 
questions remain. First, we do not know whether liquidity constraints are an important 
factor in driving the estimated effects. A second and related issue mentioned above is 
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that we do not know what returns those induced into staying on by the subsidy will 
enjoy. Finally, we really have very little idea of how these returns may change now 
that the programme has been rolled out nationally and with it the future supply of 
educated workers. This of course depends on many factors, not least the nature of the 
production function. These are all-important research and policy questions that we 
will be pursuing in the future. 
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Appendix 1: Indicators used in each deprivation score 


Income 

Adults in Income Support households (DSS) for 1998 

Children in Income Support households (DSS) for 1998 

Adults in Income Based Job Seekers Allowance households (DSS) for 1998 

Children in Income Based Job Seekers Allowance households (DSS) for 1998 

Employment 

Adults in Family Credit households (DSS) for 1999 

Children in Family Credit households (DSS) for 1999 

Adults in Disability Working Allowance households (DSS) for 1999 

Children in Disability Working Allowance households (DSS) for 1999 

Non-earning, non-IS pensioner and disabled Council Tax Benefit recipients (DSS) 

for 1998 apportioned to wards 

Unemployment claimant counts (JUVOS, ONS) average of May 1998, August 
1998, November 1998 and February 1999 

People out of work but in TEC delivered government supported training (DfEE) 
People aged 18-24 on New Deal options (ES) 

Incapacity Benefit recipients aged 16-59 (DSS) for 1998 

Severe Disablement Allowance claimants aged 16-59 (DSS) for 1999 

Health 

Comparative Mortality Ratios for men and women at ages under 65. District level 


Deprivation and figures for 1997 and 1998 applied to constituent wards (ONS) 

Disability People receiving Attendance Allowance or Disability Living Allowance (DSS) in 


Education, 

1998 as a proportion of all people 

Proportion of people of working age (16-59) receiving Incapacity Benefit or Severe 
Disablement Allowance (DSS) for 1998 and 1999 respectively 
Age and sex standardized ratio of limiting long-term illness (1991 Census) 
Proportion of births of low birth weight (<2,500g) for 1993-97 (ONS) 

Working age adults with no qualifications (3 years aggregated LFS data at district 


Skills and level, modelled to ward level) for 1995-1998 


Training 

Children aged 16 and over who are not in full-time education (Child Benefit data - 
DSS) for 1999 

Proportions of 17-19 year old population who have not successfully applied for HE 
(UCAS data) for 1997 and 1998 

KS2 primary school performance data (DfEE, converted to ward level estimates) for 
1998 

Housing 

Primary school children with English as an additional language (DfEE) for 1998 
Absenteeism at primary level (all absences, not just unauthorised) (DfEE) for 1998 
Homeless households in temporary accommodation (Local Authority HIP Returns) 
for 1997-98 

Geographical 

Household overcrowding (1991 Census) 

Poor private sector housing (modelled from 1996 English House Condition Survey 
and RESIDATA) 

Access to a post office (General Post Office Counters) for April 1998 


Access to Access to food shops (Data Consultancy) 1998 


Services 

Access to a GP (NHS, BMA, Scottish Health Service) for October 1997 
Access to a primary school (DfEE) for 1999 

Child poverty 

Percentage of children that live in families that claim means-tested benefits (Income 
Support, Job Seekers Allowance (Income Based), Family Credit and Disability 
Working Allowance). 


Source: Department of the Environment, Transport and the Regions (2001), Regeneration Research 
Summary: Indices of Deprivation 2000, (Number 31, 2000) 

( www.urban.odpm.gov.uk/research/summaries/03100/index.htm) . 
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Appendix 2: Sample characteristics 



Whole Sample 

Pilot Areas 

Control Areas 

Male 

0.504 

0.503 

0.504 

Pilot Area 

0.661 

1.000 

0.000 

Fully Eligible for EMA 

0.470 

0.472 

0.466 

Partially Eligible for EMA 

0.308 

0.308 

0.306 

Ineligible for EMA 

0.223 

0.220 

0.228 

In full-time education Year 12 

0.709 

0.717 

0.694 

In work Year 12 

0.156 

0.157 

0.154 

Characteristics used in matching 
Weekly family income 

389.01 

387.50 

391.95 

Family receives means-tested benefit 

0.263 

0.268 

0.253 

Mother and father figure present 

0.623 

0.626 

0.617 

Father figure present 

0.753 

0.753 

0.753 

Owner occupier 

0.693 

0.686 

0.709 

Council or Housing Association 

0.253 

0.266 

0.226 

Has statemented special needs 

0.092 

0.093 

0.090 

Mother’ s age 

39.859 

39.867 

39.843 

Father’ s age 

30.096 

30.301 

29.696 

Mother has A levels or higher 

0.245 

0.237 

0.259 

Mother has 0 levels or equivalent 

0.246 

0.245 

0.247 

Father has A levels or higher 

0.221 

0.220 

0.223 

Father has 0 levels or equivalent 

0.171 

0.168 

0.177 

Father manager or professional 

0.166 

0.163 

0.172 

Father clerical or similar 

0.243 

0.246 

0.238 

Mother manager or professional 

0.129 

0.121 

0.144 

Mother clerical or similar 

0.294 

0.300 

0.282 

Father variables missing 

0.363 

0.362 

0.366 

1 or 2 parents in work when born 

0.831 

0.825 

0.843 

Attended 2 primary schools 

0.254 

0.256 

0.251 

Attended more than 2 primary schools 

0.076 

0.077 

0.073 

Received childcare as a child 

0.911 

0.915 

0.903 

1 set of Grandparents around when 
child 

0.326 

0.320 

0.337 

2 sets of Grandparents around when 
child 

0.448 

0.466 

0.413 

Grandparents provided care when child 

0.316 

0.307 

0.332 

111 between 0 and 1 

0.223 

0.225 

0.219 

Number of older siblings 

0.941 

0.928 

0.968 

Number of younger siblings 

0.975 

0.979 

0.968 

Older sibling educated to 18 

0.291 

0.286 

0.299 

White 

0.896 

0.892 

0.903 

Father in full-time work 

0.503 

0.504 

0.502 

Father in part-time work 

0.021 

0.019 

0.025 

Mother in full-time work 

0.335 

0.327 

0.350 

Mother in part-time work 

0.309 

0.312 

0.304 

Maths GCSE score 

4.233 

4.232 

4.235 

English GCSE score 

3.810 

3.798 

3.834 

GCSE score missing 

0.129 

0.131 

0.126 

Number of observations 

6,838 

4,5 1 8 

2,320 
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Appendix 3: Covariate balancing indicators (best specification): before and after 
matching 

Matching N\ No Probit Probit P>j * 1 2 3 4 5 Median Median % lost 


Estimator 

pseudo 

R 2 

pseudo 

R 2 


bias 

bias 

to 

common 

support 


Before Before Before 

After 

After 

Before 

After 

After 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 


Males: 


Nearest Neighbour 

1,753 

900 

0.085 

0.029 

0.000 3.9 

5.3 

0.4 

Mahalanobis-metric 

1,753 

900 

0.085 

0.086 

0.000 3.9 

7.4 

0.4 

Epanechnikov (bw=0.01) 

1,753 

900 

0.085 

0.012 

0.740 3.9 

2.5 

0.7 

Epanechnikov (bw=0.06) 

1,753 

900 

0.085 

0.010 

0.921 3.9 

2.3 

0.4 

Females: 








Nearest Neighbour 

1,771 

891 

0.104 

0.030 

0.000 3.3 

3.6 

0.2 

Mahalanobis-metric 

1,771 

891 

0.104 

0.103 

0.000 3.3 

7.8 

0.2 

Epanechnikov (bw=0.01) 

1,771 

891 

0.104 

0.015 

0.306 3.3 

2.2 

1.0 

Epanechnikov (bw=0.06) 

1,771 

891 

0.104 

0.014 

0.510 3.3 

1.5 

0.2 


Notes: 

(1) Pseudo R 2 from probit estimation of the conditional treatment probability, giving an indication 
of how well our matching regressors X explain the relevant educational choice. 

(2) Pseudo R 2 from a probit of D on X on the matched samples, to be compared with (1). 

(3) P- value of the likelihood-ratio test after matching, testing the hypothesis that the regressors are 
jointly insignificant, i.e. well balanced in the two matched groups. 

(4) Median absolute standardised bias before and after matching, median taken over all the 

and matching. Following Rosenbaum and Rubin (1985), for a given covariate X, the standardised 

(5) difference before matching is the difference of the sample means in the full treated and non- 
treated subsamples as a percentage of the square root of the average of the sample variances in 
the full treated and non-treated groups. The standardised difference after matching is the 
difference of the sample means in the matched treated (i.e. falling within the common support) 
and matched non-treated subsamples as a percentage of the square root of the average of the 
sample variances in the full treated and non-treated groups. 


B before (x)^m 


X.-Xo 


^j(V 1 (X) + V 0 (X))/2 


B after (X) = 100 


X\M X QM 


j(V 1 (X)+V 0 (X))/2 


Note that the standardisation allows comparisons between variables X and for a given variable 
X, comparisons before and after matching. 

(6) Share of the treated group falling outside of the common support, imposed at the boundaries. 


33 


Appendix 4: Attrition between wave 1 and wave 2 

Table A4- Probability of Attrition between wave 1 and wave 2. 



Marginal 

Effect 

Standard ei 

Partially Eligible 

-0.002 

0.024 

Fully Eligible 

-0.039 

0.015 

Pilot Area 

0.005 

0.012 

Male 

0.019 

0.011 

Weekly family income 

0.000 

0.000 

Family receives means-tested benefit 

-0.014 

0.017 

Mother and Father figure present 

-0.015 

0.032 

Father figure present 

-0.028 

0.021 

Owner occupier 

-0.085 

0.025 

Council or Housing Association 

-0.031 

0.023 

Has statemented special needs 

-0.001 

0.018 

Mother’s age 

-0.002 

0.001 

Father’s age 

-0.001 

0.001 

Mother has A levels or higher 

0.001 

0.017 

Mother has O levels or equivalent 

0.001 

0.014 

Father has A levels or higher 

-0.065 

0.018 

Father has O levels or equivalent 

-0.022 

0.017 

Father manager or professional 

-0.014 

0.021 

Father clerical or similar 

0.017 

0.016 

Mother manager or professional 

-0.029 

0.020 

Mother clerical or similar 

-0.014 

0.013 

Father variables missing 

-0.015 

0.036 

1 or 2 parents in work when born 

-0.011 

0.016 

Attended 2 primary schools 

-0.021 

0.012 

Attended more than 2 primary schools 

0.030 

0.021 

Received childcare as a child 

0.002 

0.019 

1 set of Grandparents around when child 

-0.008 

0.015 

1 sets of Grandparents around when child 

0.004 

0.016 

Grandparents provided care when child 

0.007 

0.012 

111 between 0 and 1 

0.010 

0.013 

Number of older siblings 

0.017 

0.006 

Number of younger siblings 

-0.010 

0.005 

Older sibling educated to 18 

-0.036 

0.013 

White 

-0.020 

0.022 

Father in full-time work 

0.033 

0.020 

Father in part-time work 

-0.004 

0.039 

Mother in full-time work 

-0.002 

0.017 

Mother in part-time work 

-0.030 

0.015 

Income 

-0.001 

0.002 

Employment 

-0.007 

0.003 

Health Deprivation and Disability 

0.033 

0.020 

Education, Skills and Training 

0.023 

0.011 

Housing 

0.010 

0.012 

Geographical Access to Services 

0.004 

0.014 
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Child poverty 

0.002 

0.001 

per cent not staying on post 16 

-0.002 

0.001 

per cent not going to university 

-0.002 

0.002 

Class sizes in 1999 

-0.003 

0.002 

Authorised absences 

0.000 

0.004 

% getting 5 GCSE A-C in 1999 

0.001 

0.001 

% getting 0 GCSE A-G in 1999 

0.001 

0.001 

School has 6th form? 

-0.002 

0.013 

Distance to nearest year 12 provider 

0.000 

0.000 

Maths GCSE score 

-0.014 

0.006 

English GCSE score 

-0.015 

0.005 

GCSE score missing 

-0.003 

0.025 


Number of observations 6,838 

Observed probability 0.253 
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Appendix 5: Identifying assumptions and Estimation method 

Suppose the outcome of an individual with characteristics X\ who is exposed to the 
EMA is 7 1 . The same individual would have outcome 7° were she/he not to be 
exposed to the treatment. Obviously, either one or the other outcome is observed. The 
impact of the policy for the ith individual ( 7 1 - 7° ) is thus not observed. The main 
evaluation parameter that we will estimate is the impact of treatment on the treated, 
i.e. E(Y' - 7° I P = 1) , where P is one for individuals in the pilot areas and zero in 

the control areas. What we do observe is E(Y- I 7* = 1) , which is the average 
participation rate for those exposed to the EMA. To construct the counterfactual 
E(Y® I 7* =1) we assume that E(Y ]° I P t = 1 , X t ) = E(Y '° I P t = 0, X t ) which means 

that given the observable characteristics the allocation to treatment and control is 
random. Under this assumption it is now well known (see Rosenbaum and Rubin, 
1983) that we can reduce the dimension of the conditioning set from X to just 
PrlT* =11 X t ), i.e. the propensity score which is simply the probability of being 

allocated to the pilot given observed characteristics. This makes the computational 
exercise feasible and simple. Thus, given the original matching assumption we can 
also write that E(7 ( ° I P i = l,Pr(/> =11 X,.)) = E(Y l 0 I P. = 0,Pr(/> = 1 1 X i )). It follows 
that we can estimate the counterfactual by the sample analogue of 

£(7,° I P = 1 ) = E pl [E(7,° I P t = 0,Pr(^. = 1,1 X,))j , 

where E , denotes an expectation with respect to the distribution of the propensity 
score in the treatment sample. 


36 



Implementing this involves the following steps. In the first step the propensity score is 
estimated. In the second step we estimate the conditional expectation of the outcome 
in the control areas given the propensity score using a number of methods. It turns out 
that for our particular policy experiment, using an Epanechnikov kernel with 
bandwidth of 0.06 gives us the best covariate balancing indicators amongst a range of 
matching estimators that we considered. We are careful to ensure that all observations 
whose value of the propensity score is outside the range of the propensity score in the 
treatment sample are deleted. This imposes common support avoiding a major source 
of bias (see Heckman, Ichimura and Todd, 1997). Finally the overall average is 
constructed using as weights the distribution of the propensity score in the pilot areas. 
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